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(54) Moving picture decoding apparatus, control method therefor, and storage medium 
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(57) This invention provides a decoding apparatus 
and control method therefor capable of elinninating vis- 
ual flaws between frames even if several frames cannot 
be decoded. The first data is read out (S201), and de- 
coding of the frame data starts {S202). During decoding 
processing, whetherthe processing time exceeds a pre- 
determined limit time of decoding processing assigned 
to the packet is monitored (S204). If decoding process- 
ing does not end within the predetemnined limit time, on- 
ly decoded data are written in a memory (S213). This 
can maintain image continuity. 
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Description 

[0001] The present invention relates to an apparatus of decoding a moving picture, a control method therefor and 
a storage medium. 

[0002] As a color stilt picture encoding method, JPEG, an international standard, has widely been known As a com- 
pression method having a larger number of functions than JPEG, a new encoding method based on wavelet transfomi 
i.e., so-called JPEG 2000 is being standardized. 

[0003] Further, attempts are made to apply the wavelet transfomi to moving picture encoding. 
[0004] As the processing speed of recent CPUs increases, processing of a moving picture compressed using the 
wavelet transform technique can be realized. In a moving picture, data are successive so decoding processing must 
be completed within a predetemnined time. 

[0005] The present invention has been made to overcome the conventional drawbacks, and has as an object to at 
least reduce visible flaws between frames even if several frames cannot be decoded. 

[0006] According to the present invention, there is provided a decoding apparatus for decoding an input signal which 
includes a code stream and data having time management information about decoding of the code stream comprising- 
buffenng means for buffering the input signal; decoding means for reading out the input signal from the buffering means' 
decoding the input signal, and writing decoded data in a predetermined memory; output means for outputting the 
decoded data from the predetermined memory; and control means for controlling the buffering means and the decoding 
means on the basis of the lime management infonnalion. 

[0007] According to another aspect of the present invention, there is provided a method of controlling a decoding 
apparatus for decoding an input signal which includes a code stream and data having time management infonnation 
about decoding of the code stream, comprising: a buffering step of buffering the input signal in a predetennined buffer- 
a decoding step of reading out the input signal from the predetermined buffer, decoding the input signal and writing 
decoded data in a prcdctcmninod memory; a output step of outputting the decoded data from the predetennined mem- 
ory; and the control step of controlling at least one of the buffering step, the decoding step, and the output step on the 
basis of the time management infomnation. 

[0008] In still another aspect of the present invention, there is provided a storage medium which stores program 
codes functioning as a decoding apparatus for decoding an input signal which includes a code stream and data having 
time management infonnation about decoding of the code stream, comprising: a program code of the buffering step 
of buffering the input signal in a predetermined buffer; a program code of the decoding step of reading out the input 
signal from the predetermined buffer, decoding the input signal, and writing decoded data in a predetermined memory- 
a program code of the output step of outputting the decoded data from the predetermined memory: and a program 
code of the control step of controlling at least one of the buffering step, the decoding step, and the output step on the 
basis of the time management infonnation, 

[0009] Other features and advantages of the present invention will be apparent from the following description taken 
in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts 
throughout the figures thereof. 

[0010] The accompanying drawings, which are incorporated in and constitute a part of the specification illustrate 
embodiments of the invention and. together with the description, serve to explain the principles of the invention bv wav 
of example only. ^ ' 

Fig. 1 is a block diagram showing the schematic arrangement of a decoding apparatus in the first embodiment of 
the present invention; 

Fig. 2 is a flow chart showing decoding processing of the decoding apparatus in the first embodiment of the present 
invention; 

Fig. 3A is a view showing decoding processing in 2x-speed playback according to the first embodiment of the 
present invention; 

Fig. 3B is a view showing an example of an image displayed in nomnal playback according to the first embodiment 
of the present invention; 

Fig. 3C is a view showing an example of an image displayed in 2x-speed playback according to the first embodiment 
of the present invention; 

Fig. 4A is a view showing an entire encoded stream, which should bo originally decoded in the first embodiment 
of the present invention; 

Fig. 4B is a view showing data decoded partway owing to a short processing time in the first embodiment of the 
present invention; 

Fig. 5 is a flow chart showing decoding processing of a decoding apparatus in the second embodiment of the 
present invention; 

Fig. 6A is a view showing decoding processing in 4x-speed playback according to the second embodiment of the 
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present invention: 

Fig. 6B is a view showing an example of an image displayed in 4x-speed playback according to the second em- 
bodiment of the present invention; 

Fig. 7 is a block diagram showing the arrangement of a conventional encoder: 

Fig. 8A is a block diagram showing the arrangement of a discrete wavelet transform unit 702; 

Fig. 8B is a block diagram showing the arrangement of a processing unit 802 in Fig. 8A; 

Fig. 8C is a view showing the layout of 2-level transform coefficient groups obtained by two-dimensional transform 
processing; 

Fig. 9 is a view for explaining the operation of an entropy encoding unit 704; 
Fig. 1 0A is a view showing the structure of an entire code stream; 
Fig. 1 0B is a view showing a structure of a main header MH; 
Fig. 10C is a view showing a structure of a tile header TH; 
Fig. 10D is a view showing a structure of a bit stream; 

Fig. 11 A is a view showing the entire structure of a code stream different from that of Fig. 10A; 
Fig. 11 B is a view showing a structure of the main header MH; 
Fig. lie is a view showing a structure of the tile header TH; 
Fig. 11 D is a view showing a structure of the bit stream; 

Fig. 12 is a block diagram showing the arrangement of a conventional decoder; 
Fig. 13 Is a view for explaining the operation of an entropy decoding unil 1202; 

Fig. 14A is a block diagram showing the arrangement of an inverse discrete wavelet transform unit 1204; 
Fig. 14B is a block diagram showing the arrangement of a processing unit 1402 in Fig. 14A; 
Fig. 15A is a view showing a structure of a code stream; 

Fig. 15B is a view showing the relationship between each subband and the size of a displayed image; 
Fig. 16A is a view showing a structure of a code stream different from that of Fig. ISA; 
Fig. 16B is a view showing an example of a displayed image; 

Fig. 17 is a block diagram showing a conventional decoding apparatus of decoding multiplexed video and audio 
data; 

Fig. 18A is a view for explaining conventional processing from reception to display of a packet; 

Fig. 1 8B is a view for explaining conventional processing from reception to display of a packet in 2x-speed playback; 

Fig. 19A is a view for explaining decoding processing in the second embodiment of the present Invention; 

Fig. 1 9B is a view showing an example of an image displayed according to the second embodiment of the present . 

invention; 

Fig. 20 Is a flow chart showing decoding processing of a decoding processing in the third embodiment of the 
present Invention; 

Fig. 21 A is a view showing an example of dividing one frame into nine tiles in the third embodiment of the present 
invention; 

Figs. 21 B to 21 J are views showing examples of shuffling tiles according to the third embodiment of the present 
invention; 

Fig. 22 is a graph for explaining the relationship between the Image quality and the data amount; 

Fig. 23 Is a flow chart showing decoding processing of a decoding apparatus in the fourth embodiment of the 

present Invention; 

Fig. 24A is a view for explaining decoding processing by 2x-speed playback in the fourth embodiment of the present 
invention; 

Fig. 24B is a view showing an example of an image displayed in normal playback according to the fourth embod- 
iment of the present invention; 

Figs. 24C and 24D are views showing examples of an image displayed In 2x-speed playback according to the 
fourth embodiment of the present invention; 

Fig. 25A is a view showing an entire encoded stream which should be originally decoded in the fourth embodiment 
of the present invention; 

Fig. 25B is a view showing data decoded partway owing to a short processing time in the fourth embodiment of 
the present invention; 

Fig. 26 is a flow chart showing decoding processing of a decoding apparatus in the fifth embodiment of the present 
invention; 

Fig. 27A is a view for explaining decoding processing by 4x-speed playback in the fourth embodiment of the present 
invention; 

Fig. 27B is a view showing an example of an image displayed in 4x-speed playback according to the fourth em- 
bodiment of the present invention; 

Fig. 28A is a view for explaining decoding processing by 4x-speed playback in the fifth embodiment of the present 
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invention; 

Fig. 28B is a view showing an example of an image displayed in 4x-speed playback according to the fifth embod- 
iment of the present invention; 

Fig. 29 is a flow chart showing decoding processing of a decoding apparatus in the sixth embodiment of the present 
invention; 

Fig. 30 is a graph for explaining the relationship between the image size and the data amount; 
Fig. 31 is a view for explaining the relationship between the frame rate and the data amount; 
Fig. 32 is a table showing an example of a table representing the relationship between the playback speed of high- 
speed playback and the lowest image quality in the second embodiment of the present invention; 
Fig. 33 is a table showing an example of a table representing the relationship between the playback speed of high- 
speed playback and the smallest image size in the fifth embodiment of the present invention; and 
Fig. 34 is a block diagram showing an example of a personal computer functioning as an embodiment of a decoding 
apparatus of the present invention. 



(001 1 ] Preferred embodiments of the present invention will be described in detail below with reference to the accom- 
panying drawings. Program codes complying with a flow chart described in each embodiment are stored in a memory 
formed from a RAM or ROM in a decoding apparatus according to the embodiment, and read out and executed bv a 



20 [First Embodiment] 

[0012] A conventional encoding method based on the wavelet transfomri will be explained. 
(Encoder) 

25 

[0013] In Fig. 7 which is a block diagram showing a conventional encoder, reference numeral 701 denotes an image 
input unit; 702, a discrete wavelet transform (DWT) unit; 703. a quantization unit; 704, an entropy encoding unit; and 
705, a code output unit. 

[0014] Pixel signals fomrting an image to be encoded are input to the image input unit 701 in a raster scan order and 
30 an output from the image input unit 701 is input to the discrete wavelet transfomn unit 702. In the following description, 
an image signal expresses a monochrome multilevel image. To encode a plurality of color components, like a color 
image, R. G, and B color components or luminance and chromaticity components are compressed as a single color 
component. 

[0015] The discrete wavelet transform unit 702 perfonns two-dimensional discrete wavelet transform processing for 
35 an input image signal, and cateulates and outputs a transfonn coefficient. Fig. 8A shows the basic arrangement of the 
discrete wavelet transform unit 702 in which input image signals are stored in a memory 801 , sequentially read out 
and subjected to transfomn processing by a processing unit 802, and rewritten in the memory 801 . The processing unit 
802 has a processing arrangement shown in Fig. 88. In Fig. 8B, the input image signal is separated into even- and 
odd-numbered address signals by a combination of a delay element and down samplers, and subjected to filter process- 
40 ing by two filters p and u. Reference symbols s and d in Fig. 8B denote low- and high-pass coefficients in 1 -level 
decomposition for one-dimensional image signals, and are calculated by 



45 



50 



55 



d(n) = x(2*n+1) - floor((x(2*n)+x(2*n+2))/2) (1) 

s(n) = x(2*n) + f loor((d(2-1 )+d(n))/4) (2) 

where x(n) is the image signal to be transfonned, and floor(r) is a maximum integer value smaller than r 
[0016] By the above processing, image signals undergo one-dimensional discrete wavelet transfonri processing. 
The two-dimensional discrete wavelet transform sequontially executes a one-dimcnsional transfonn in the horizontal 
and vertical directions of an image. Details of the two-dimensional discrete wavelet transfonn are well known, and a 
description thereof will be omitted. 

[0017] Fig. 8C shows the layout of 2-level transfonn coefficient groups obtained by two-dimensional transform 
processing. An image signal is decomposed into coefficient streams HH1 , HL1 . LH1 LL having different frequency 
bands. In the following description, these coefficient streams will be called subbands. The coefficient of each subband 
is output to the subsequent quantization unit 703. 

[0018] The quantization unit 703 quantizes the input coefficient at a predetemnined quantization step, and outputs 
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an index (quantisation index) for the quantised value. The quantisation unit 703 executes quantisation by 



q = sign(c)tloor(abs(c)/A) 



(3) 



5 



sign(c) = 1 ;c >= 0 



(4) 



10 



sign(c) - -1;c <- 0 



(5) 



where c is a coefficient to be quantized, and floor(X) is a maxinnum integer value snnaller than X. The A value can be 
1 . In this case, no quantization is actually done, and a transform coefficient input to the quantization unit 703 is directly 
output to the subsequent entropy encoding unit 704. 
15 [0019] The entropy encoding unit 704 decomposes the input quantization Index Into bit planes, performs binary 
arithmetic encoding in units of bit planes, and outputs a code stream. 

[0020] Fig. 9 is a view for explaining the operation of the entropy encoding unit 704. In this example, three nonzero 
quantisation indices exist in the regions of a 4 x 4 subband, and have values of +13, -6, and +3, respectively. The 
entropy encoding unit 704 scans the respective subband regions to obtain a maximum value M among all the quanti- 
20 zation Indices, and calculates the number S of bits necessary for expressing the maximum quantization index by 



25 where ceil(x) is a minimum integer value among integers equal to or larger than x. In Fig. 9, the maximum quantization 
index value M is 13. From equation (6), S is 4. and 16 quantization indices in a sequence arc processed in units of 
four bit planes, as shown In Fig. 9. 

[0021] The entropy encoding unit 704 executes binary arithmetic encoding for the respective bits of the most signif- 
icant bit plane (represented by MSB in Fig. 9), and outputs them as a bit stream. The entropy encoding unit 704 switches 
30 the bit plane to one lov/er by one level, and then encodes the respective bits of the bit plane and outputs them to the 
code output unit 705 until the target bit plane reaches the least significant bit plane (represented by LSB in Fig. 9). At 
this time, the code of each quantization index is entropy-encoded immediately after a nonzero bit Is first detected in 
bit plane scan. 

[0022] Fig. 1 0A is a schematic view showing the structure of a code stream generated and output in this manner. In 
35 Fig. 10A, MH represents a main header; TH, a tile header; and BS, a bit stream. As shown in Fig. 1 0B, the main header 
MH is constituted by pieces of infonmation including the size of an image to be encoded (the numbers of pixels in the 
horizontal and vertical directions), the size of a tile obtained by dividing the image into a plurality of rectangular regions, 
the number of components representing the number of color components, and component information representing 
the size of each component and the bit precision. When the image is not divided into tiles, the tile and image sizes 
take the same values. When the target image is a monochrome multilevel image, the number of components is 1 . 
[0023] Fig. 10C shows the structure of the tile header TH. The tile header TH is constituted by pieces of infonmation 
representing a tile length including the bit stream length and header length of a tile, and an encoding parameter for 
the tile. The encoding parameter includes the level of discrete wavelet transfomn and the type of filter 
[0024] Fig. 10D shows the structure of the bit stream. In Fig. 1 0D, bit streams are gathered in units of subbands and 
arranged in an ascending order of resolution from the bit stream of a subband whose resolution is the lowest. Bit planes 
are arranged from an upper bit plane to a lower bit plane in each subband, and codes are arranged in each bit plane. 
[0025] Fig. 11 A shows the entire structure of a code stream different from that of Fig. 10A. MH represents a main 
( header; TH, a tile header; and BS. a bit stream. As shown in Fig. 1 1 B, the main header MH is constituted by pieces of 
information including the size of an image lo be encoded (the numbers of pixels in the horizontal and vertical directions), 
the size of a tile obtained by dividing the Image into a plurality of rectangular regions, the number of components 
representing the number of color components, and component information representing the size of each component 
and the bit precision. When the image is not divided into tiles, the tile and image sizes take the same values. When 
the target image is a monochrome multilevel image, the number of components is 1 . 

[0026] Fig. 11 C shows the structure of the tile header TH. The tile header TH is constituted by pieces of information 
55 representing a tile length including the bit stream length and header length of a tile, and an encoding parameter for 
the tile. The encoding parameter includes the level of discrete wavelet transform and the type of filter. Fig. 1 1 D shows 
the stnjcture of the bit stream. In Fig. 11 D, bit streams are gathered In units of bit planes and arranged from an upper 



S = ceil(log2(abs(M)) 



(6) 
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bit plane to a lower bit plane. In each bit plane, the encoding results of the bit planes of quanti7ation indices in each 
subband are sequentially arranged in units of subbands. In Fig. 11D. S represents the number of bits necessary for 
expressing the maximum quantization index. A code stream generated in this manner is output to the code output unit 
705. 

[0027] The compression ratio of an entire image to be encoded can be controlled by changing the quantization step 
A. As another method, the entropy encoding unit 704 can also control (discard) the lower bit of a bit plane to be encoded 
in accordance with a necessary compression ratio. In this case, all the bit planes are not encoded, bit planes corre- 
sponding in number to a desired compression ratio are encoded from an upper bit plane, and the encoded bit planes 
are Included in the final code stream. 

(Decoder) 

[0028] A method of decoding a code stream attained by the above-described encoder will be explained. 
[0029] Fig. 12 is a block diagram showing the arrangement of the decoder. Reference numeral 1201 denotes a code 
input unit; 1202, an entropy decoding unit; 1203, an inverse quantization unit; 1204, an inverse discrete wavelet trans- 
form unit; and 1205, an image output unit. 

[0030] The code input unit 1201 receives the above-described code stream, analyzes the header included in it 
extracts a parameter necessary for subsequent processing, and if necessary, controls the processing flow or outputs 
the parameter to the subsequent processing unit. The bit stream included in the code stream is output to the enlroov 
20 decoding unit 1202. 

[0031] The entropy decoding unit 1202 decodes the bit stream in units of bit planes, and outputs the decoded stream 
This decoding procedure is shown in Fig. 13. Fig. 13 shows the flow of sequentially decoding one region of a subband 
to be decoded in units of bit planes and finally decoding the quantization index. Bit planes are decoded in the direction 
indicated by an arrow in Fig. 13. The decoded quantization index is output to the inverse quantization unit 1203. 
25 [0032] The inverse quantization unit 1203 reconstructsthediscrete wavelet transform coefficient from the input quan- 
tization Index by: 



10 



15 



30 



35 



40 



C - A*q ; q ;t 0 (7) 



c' = 0;q = O (8) 



where q is the quantization index. A is the quantization step and has the same value as that used In encoding and C 
IS the decoded transfomn coefficient obtained by decoding the coefficient s or d In encoding. The transform coefficient 
C Is output to the subsequent inverse discrete wavelet transfonn unit 1204. 

[0033] Fig. 14A Is a block diagram showing the an-angement of the Inverse discrete wavelet transform unit 1 204 In 
Fig. 14A, the Input transform coefficient Is stored in a memory 1401 . A processing unit 1402 executes the two-dimen- 
sional inverse discrete wavelet transform by perfomning a one-dimensional inverse discrete wavelet transform se- 
quentially reading out transfonn coefficients from the memory 1 401 , and processing them. The two-dimensional Inverse 
discrete wavelet transfomn is executed by procedures reverse to those of the fonward transform. Details of the two- 
dimensional Inverse discrete wavelet transfomn are well known, and a description thereof will be omitted. Fig. 14B 
shows the processing block of the processing unit 1 402. Input transfomn coefficients are subjected to two filter processes 
by u and p. up-sampled, and superimposed on each other, thereby outputting an Image signal x'. These processes 
are perfomned by 



so 



55 



X' (2*n) = s'(n) - floor((d'(n-1)-Ki'(n))/4) (9) 

x' (2*n+1) = d'(n) +floor((x'(2*n)+x'(2*n+2))/2) (10) 

[0034] In this case, the fonward and backward discrete wavelet transfomn processes by equations (1), (2) (9) and 

(10) satisfy perfect reconstruction conditions. Thus, the decoded image signal x' coincides with the signal x of the 

onginal image as far as the quantization step A is 1 and all the bit planes are decoded In bit plane decoding 

[0035] By this processing, the image is restored and output to the image output unit 1205. The image output unit 

1205 may be an image display device such as a monitor or a storage device such as a magnetic disk, 

[0036] The image display form in restoring and displaying an image by the above-mentioned procedures will be 
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described with reference to Figs. 15A and 15B. Fig. 15A shows an example of a code streann whose basic structure 
is based on Figs. 10A and 10D- When an entire image is set as one tile, the code stream includes only one tile header 
and one bit stream. As shown in Fig. ISA, codes are arranged in a bit stream BSO in an ascending order of resolution 
from LL which is a subband corresponding to the lowest resolution. 

5 [0037] The decoder sequentially reads bit streams, and when decoding codes corresponding to each subband. dis- 
plays the image. Fig. 15B shows the correspondence between each subband and the size of a displayed image. In 
this example, the two-dimensional discrete wavelet transform is done to two levels. When only LL is decoded and 
displayed, an image made up of pixels reduced in number to 1/4 the original image in the horizontal and vertical 
directions is restored. When bit streams are further read and all the subbands of level 2 are restored and displayed, 

10 an image made up of pixels reduced In number to 1/2 the original image in the respective directions is restored. When 
all the subbands of level 1 are restored and displayed, an image having the same number of pixels as that of the 
original image is restored. 

[0038] Another image display form in restoring and displaying an image by the above-mentioned procedures will be 
described with reference to Figs. 16A and 168. Fig. 16A shows an example of a code stream whose basic structure 
IS is based on Figs. 1 1 A and 1 1 D. When an entire image is set as one tile, the code stream includes only one tile header 
and one bit stream. As shown in Fig. 1 6A. codes are arranged from the most significant bit plane to the least significant 
bit plane. 

[0039] The decoder sequentially reads bit streams, and when decoding the codes of each bit plane, displays the 
image. Fig. 16B shows an example of a change in the quality of a displayed image when decoding is sequentially done 

20 from an upper bit plane. When only the upper bit plane is decoded, only the feature of the entire image is displayed. 
As lower bit planes are decoded, the image quality is improved stepwise. When the quantization step A is 1 in quan- 
tization, the image displayed after all the bit planes are decoded is completely the same as the original image. 
[0040] In this prior art, lower bit planes to be decoded by the entropy decoding unit 1202 can be limited (ignored) to 
decrease the encoded data amount to be received or processed, thereby controlling the compression ratio. This enables 

25 obtaining a decoded image having a desired image quality from only a necessary amount of encoded data. The prior 
art can also realize reversible encoding/decoding in which a restored image coincides with its original image when the 
quantization step A is 1 in quantization and all the bit planes are decoded in decoding. 

[0041] Fig. 17 shows a conventional decoding apparatus of decoding multiplexed video and audio data. A OEMUX 
unit 1701 demultiplexes video and audio streams. The data unit output to each of encoding buffers 1702a and 1702b 
30 is a sync processing unit generally called an access unit. Letting a frame be a processing unit, the processing proceeds 
such that the encoding buffers 1702a and 1702b receive data in units of frames, decoding units 1703a and 1703b 
decode the data, the decoded data are written in memories 1 704a and 1 704b, and output units 1 705a and 1 705b 
display the data. The access unit includes, at the header, time management information, which is used for sync man- 
agement. 

35 [0042] Fig. 18A shows timings from reception to display of a packet. The first packet received at timing t = tOO takes 
a time D1 for decoding processing and is displayed for a time CI . Similarly, the second packet received at timing t = 
t40 takes a time D2 for decoding processing and is displayed for a time C2. If decoding processing ends within a 
predetermined time, the same data as that on the encoding side is decoded with a predetermined delay time. For 
descriptive convenience, the buffering size is one frame. Processing must end till the read timing of the next frame 

'40 data, which is the decoding limit time. In this case, the limit time described at the header is (t40 - tOO), and the condition 
of displaying the first frame is D1 ^ (t40-tOO). In general, the processing time is designed to be much shorter than the 
limit time. However, when decoding is done by software installed in a general-purpose computer, 100% of the CPU 
pertomnance cannot always be assigned to this processing due to execution of another application. To read data once 
stored in a hard disk at a high speed, 2x-speed playback requires two or more times the processing speed, and 3x- 
speed playback requires three or more times the processing speed. At this time, time management information within 
a packet header must be rewritten in accordance with the playback speed, and then processing is executed. 
[0043] A problem when the processing speed is insufficient will be explained with reference to Fig. 1 SB by exempli- 
fying 2x-speed playback. The first packet received at timing t = tOO takes the time D1 for decoding processing and is 
displayed for the time CI . At this time, the processing limit time for the first packet is {t20-t00) or (t40-t00)/2. Since D1 

^0 ^ (t40-t00), the first packet can be displayed. However, as for the second packet received at timing t - t20, the time 
D2 for decoding processing is D2 > {t40-t20), so decoding processing of the next packet must start before decoding 
processing of the second packet ends. In this case, data which is being decoded is not written in the memory and holds 
its state until the next write is done. That is, the first data is kept displayed, and the display time is the time C2 during 
which the second data should be displayed. The third packet received at timing t = t40 takes the time D3 for decoding 

55 processing. The result can be displayed because D3 ^ (t60-t40), and the third packet is displayed for a time C3. As 
for the fourth packet, the result can be displayed because D4 < (t80-t40), and the fourth packet is displayed for a time 
C4. 

[0044] In conventional decoding processing, if decoding cannot be completed owing to high-speed search, discon- 
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tinuity is generated between frames, and visual flaws occur in the time axis direction. A decoding apparatus of the first 
embodiment for solving this problem will be described below. 

[0045] Fig. 1 is a block diagram showing the schematic an-angement of the decoding apparatus in the first embod- 
iment. Reference numeral 101 denotes a DEMUX unit; 102a and 102b. decoding buffers: 103a and 103b decoding 
units; 104a and 104b, memories; 105a and 105b. output units; and 106a and 106b, sync controllers. The units 102a 
to 106a (to be referred to as decoder 1 ) and the units 102b to 106b (to be refen-ed to as decoder 2) have the same 
an-angement, which means that a plurality of decoding processes are performed parallel. Generally, video decoding 
and audio decoding correspond to these decoding processes. One of the features of the first embodiment is that 
decoders 1 and 2 adopt the sync controllers. Decoder 1 will be explained mainly for the sync controller. Note that it is 
easily understood that the following description also applies to decoder 2. 

[0046] The sync controller 106a detects the times at which the header of an access unit should be read and decoding 
1 should end processing. If decoder 1 cannot complete processing within a necessary processing time the following 
control is done for the decoding buffer 1 02a and decoding unit 1 03a. The timing of control and its processing contents 
will be described with reference to Fig. 2 showing the flow chart of this processing. In this case, the access unit Is a frame 
[0047] The first data is read in step S201 , and decoding of the frame data starts in step S202. This processing means 
a state wherein the decoding unit 1 03a reads out data from the decoding buffer 102a on demand and decodes them 
in Fig. 1 . Until decoding processing ends at the branch in step S203. whether the current time exceeds the processing 
limit time assigned to this packet Is monitored In step S204. For convenience, the time till the reception timing of the 
next frame may be assumed as an assigned maximum decoding processing lime. Within the assigned processing limit 
time, the current data is kept decoded. If a bit stream is SNR scalable, respective subband data are an-anged from the 
low-frequency side to the high-frequency side for each bit plane. An example is bit planes included in the code stream 
shown in Fig. 1 6A. If the number of bits assigned to each subband changes, the number of subbands on the MSB side 
is smaller than that on the LSB side. In this example, the number of subbands on the MSB side is three. LL, HL2. and 
LH2. The first current data in Fig. 2 is an LL band of a bit piano (S-1 ). 

[0048] After step S205, data of the next subband. i.e.. HL2 is set as new data to be processed in step S206 The 
flow passes through this loop again to process LH2, and decoding of all the data included in the bit plane (S-1) ends 
After step S207, the flow shifts to processing of a bit plane BitsS-2 in step S208. The flow repeats this loop to end 
decoding up to a bit plane 0 and end decoding of all the data of this frame. 

[0049] In step S209, the current frame data is written in the memory. The display timing complies with another infor- 
mation in the packet header. At the reception timing of the next frame data in step 321 0, whether the next frame data 
exists Is checked in step S21 1 . If YES in step S21 1 , the next frame data is set as current frame data in step S21 2 and 
the flow returns to processing of step S202; if NO in step S211 . a series of processes end. If it is detennined in step 
8204 that processing did not end within the limit time, only decoded data are written in the memory in step S213 The 
display timing complies with time management infomriation in the packet header. Since decoding processing ends 
within the limit time, the display time defined In the header coincides with the actual data display time. This relationship 
will be explained with reference to Figs. 3A to 3C. 

[0050] Fig. 3A shows 2x-speed playback, similar to Fig. 1 SB. The first packet received at timing t = tOO takes a time 
D1 for decoding processing and is displayed for a time CI . The second packet received at timing t = t20 takes a time 
D2 for decoding processing, but all the data cannot be decoded within this time because D2 > (t40-t20) Fig 4A shows 
an entire encoded stream which should be originally decoded, and Fig. 4B shows data decoded partway owing to a 
short processing time. In this example, the final bit plane BitO, and HL1, LH1 and HH1 of the bit plane BItl are not 
decoded. Hence, the image of this frame is slightly lower in image quality than other frames. Fig. 3C is a view for 
explaining the image quality in 2x-speed playback. The image quality of the second frame is slightly lower than the 
image quality in nornial playback in Fig. SB. However, the number of frames to be played back does not decrease so 
that a smooth motion can be reproduced similarly to normal playback in Fig. 3B. In this manner, if the processing time 
IS short, only decoded data are written in the memory to guarantee the image quality of at least the decoded data 
[0051] Fig. 6A shows 4x-speed playback. As for the first packet received at liming I = tOO. all the data cannot be 
decoded within the time D1. and only data which can be decoded till the next packet reception timing t = tlO are 
displayed for the time CI, This also applies to D2, D3. and D4. In this example, as all the four packets cannot be 
decoded within their processing times, the images of the frames are displayed at a low image quality However the 
mrl^of ^'^""^^ *° ^^""^ "^"^^ decrease even at a 4x speed, so that a smooth motion can be reproduced 

[0052] As described above, image continuity can be maintained by the above processing for each packet thouqh 
the image quality slightly changes between frames. 

[Second Embodiment] 

[0053] In the second embodiment, even a frame whose processing time cannot fall within a predetermined time 
maintains image quality at a predetemnined or higher level. A decoding apparatus in the second embodiment has the 
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same arrangement as that in Fig. 1 . 

[0054] The flow chart of decoding processing of the decoding apparatus in the second embodiment will be described 
with reference to Fig. 5. Note that a description of the same processing as in the flow chart shown in Fig. 2 will be 
omitted. Assuming that processing of step S201 has already ended in processing of step S301. processing of step 

5 S201 Is not shown in Fig. 5. 

[0055] If the end of decoding of the current data is determined In step S302. the same processing as processing 
from steps S205 to S211 Is executed. Whether the processing time has reached the limit time Is checked in step S303. 
If YES in step S303, whether the image quality of the decoded data is higher than the lowest image quality set In 
advance is determined in step S304. The Image quality is set In advance such that alt data must be decoded up to the 

10 bit plane (5-1) or data must be decoded up to the most significant bit plane of all the subbands. The lowest Image 
quality is preferably set In accordance with the processing perfonmance of the decoding unit, which will be described 
later. If YES in step S304, the flow advances to step S305, and the decoding result is written in a memory 104a. 
[0056] Similar to Fig. 2, the flow waits for the reception timing of the next frame data in step S306. If the next frame 
data exists in step S307, the next frame data Is set as current frame data In step S308, and decoding processing 

'5 continues. If NO In step 8307, the current frame data is determined as a final frame, and a series of processes end. 
[0057] If NO In step S304, the flow waits for the reception timing of the next frame data in step 8309. If the next 
frame data exists in step 8310, the limit time is changed in step 8311 . The limit time is changed by newly adding time 
management infonmatlon within the packet header. That is, the current frame is processed within the total time of the 
processing lime of the current frame and the processing lime of the next frame. In step 8312, the next frame data Is 

20 discarded, and data In a decoding buffer 1 02a Is held. 

[0058] This relationship will be explained by exemplifying 4x-speed playback in Fig. 19A. Fig. 19B shows a display 
example of an Image In this case. In decoding processing of the first frame in Fig. 19A, all encoded data cannot be 
decoded, but the image quality is higher than the lowest image quality, and the first frame image is displayed for the 
time C1 . In decoding processing of the second frame, data cannot be decoded even with the lowest image quality 

25 within the limit time D2. so that the decoding time of the second frame is assigned up to the decoding limit time t of the 
third frame t30. The display time C2 of the second frame Image corresponds to two frames. Hence, the third frame 
Image Is not displayed, but the Image quality of the second frame Image is held. 

[0059] When the lowest image quality is not guaranteed, the frame frequency (the number of frames displayed per 
second), i.e.. the display frame rate increases as the speed of high-speed playback Increases. In some cases, the 

30 image quality is preferably improved even by decimating or down-sampling frames in terms of the processing perfomi- 
ance of the decoding unit, the display perfomnance of the display, and visual effects. Fig. 31 Is a graph showing an 
example of the relationship between the data amount per frame and the display frame rate. To Increase the display 
frame rate, the data amount per frame must be decreased. To Increase the data amount per frame, the display frame 
rate must be decreased. The data amount and image quality have a relationship as shown In Fig. 22. In this case, 

35 decreasing the data amount per frame degrades the image quality. The lowest Image quality is set by balancing the 
smoothness of the frame motion and the image quality of each frame. As a simple implementation means, the rela- 
tionship between the playback speed of high-speed playback and the lowest image quality is held as a table. Fig. 32 
shows an example of the table. This example assumes data of 8-bit precision. All the bits are guaranteed up to a 2x 
speed, upper six bits are guaranteed up to a 4x speed, and upper four bits are guaranteed up to a 6x speed. For a 

40 speed higher than the 6x speed, an Image quality of upper three bits is guaranteed. 

[0060] As described above, the decoding apparatus of the second embodiment always manages the image quality 
of a frame to be decoded and always generates an image with a predetermined or higher Image quality. 

[Third Embodiment] 

45 

[0061] A decoding apparatus in the third embodiment has the same an'angement as that in Fig. 1 , but has different 
processing control contents. Details of the processing will be described with reference to the flow chart of decoding 
processing in the third embodiment shown in Fig. 20. A description of the same processing as that in the flow chart 
shown in Fig. 2 will be omitted. In the case, the processing unit Is not a frame but a tile. The tile is obtained by breaking 

50 up a frame into a plurality of regions, and processing for each region is repeated by the number of tiles. 

[0062] The first plane of the first tile In the current frame is processed for each band (loop of steps S2003 — > S2004). 
After processing of ail the bands ends, processing is repeated for the next plane (loop of steps 82005 — > 82006). After 
processing of all the planes ends^ processing is repeated for the next tile (loop of steps S2007 ^ 82008). This process- 
ing is repeated for all the tiles in the frame. 

55 [0063] If the lowest Image quality in high-speed playback is set In processing In units of tiles, a region where the 
image is updated may be localized owing to the difference In data amount between tiles on the screen. This is prevented 
by setting the next tile data at random in step 82008, which will be explained with reference to Figs. 21 A to 21J. 
[0064] Fig. 21 A shows an example of dividing one frame Into nine tiles. Figs. 21 B to 21 J show shuffling examples 
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Of nine patterns using these tiles. The decoding priority is T1 T2 ... TP. In shuffling, the priority is sequentially 
changed at the same position of each tile on the screen. When decoding processing cannot be completed and data 
T9 fails to be decoded, data below the center in Fig. 21 B, upper right data in Fig. 2^C, and data left to the center in 
Fig. 21 D cannot be decoded. Accordingly, successive failure in decoding data at the same position can be prevented. 
[0065] When only a tile T1 can be processed by the decoder within a given time, the display is updated at a position 
above the center in Fig. 21 a position below the center in Fig. 21 C. and an upper right position in Fig. 21 D. In this 
way. data is updated at different positions for respective frames, and data goes the round by nine frames. Shuffling is 
implemented by reading the header of each packet and setting the decoding start position of the decoding unit in the 
decoding buffer. At this time, the read position of bit stream data and the write position of decoded data in the memory 
10 are made to coincide with each other. 



[0066] As described above, decoding processing of the decoding apparatus according to the third embodiment can 
prevent the image at the same position from failing in decoding successively because the tile position not decoded 
changes in playing back a decoded frame. 

'5 [Fourth Embodiment] 

[0067] A decoding apparatus in the fourth embodiment has the same arrangement as that in Fig. 1 , but has different 
processing control contents. The control timing and processing contents are shown in the flow chart of Fig. 23. In this 
embodiment, the access unit is a frame. 

[0068] The first data is read in step S2301 , and decoding of the frame data starts in step S2302. This processing 
means that a decoding unit 1 03a or 1 03b in Fig. 1 reads out data from a decoding buffer 1 02a or 1 02b on demand and 
decodes the data. 

[0069] Until decoding processing ends at the branch in step S2303, whether the current time exceeds the processing 
limit time assigned to this packet is monitored in stop S2304. For convenience, the time till the reception timing of the 
next frame may be assumed as an assigned maximum decoding processing time. Within the assigned processing limit 
time, the current data is kept decoded. If a bit stream is space-scalable, respective bit plane data are arranged from 
the upper-bit side to the lower-bit side for each subband. An example is subbands Included in the code stream shown 
in Fig. 15A. 

[0070] If the end of decoding of the current data is detemnined in step S2303, the flow advances to step S2305. and 
whether all the planes have been processed is checked. After step S2305, the next plane data, i.e., BitS-2 is set as 
new data to be processed In step S2306. After this loop is done for all the planes, processing for the first subband LL 
ends. After step S2307 of detemiining whether all the bands have been processed, processing shifts to the next sub- 
band HL2 in step S2308. After the above loop is repeated and decoding proceeds up to the final subband HH1 , all the 
data of the frame have been decoded. 

[0071] In step S2309. the current frame data is written in the memory. The display timing complies with another 
infonnation in the packet header. At the reception timing of the next frame data in step S231 0, the flow advances to 
step S231 1 , and whether the next frame data exists is checked in step S231 1 . If Yes in step S231 1 . the next frame 
data is set as current frame data in step S231 2, and the flow returns to processing of step S2302; if No in step S231 1 
a series of processes end. 

[0072] If it is determined in step S2304 that processing did not end within the limit time, the flow shifts to step S2313 
and only decoded data are written in the memory. The display timing complies with time management information iri 
the packet header. Since decoding processing ends within the limit time, the display time defined in the header coincides 
with the actual data display time. This relationship will be explained with reference to Figs. 24A to 24D. 
[0073] Fig. 25A shows 2x-speed playback, similar to Fig. 1 SB. The first packet received at timing t = tOO takes the 
time D1 for decoding processing and is displayed for the time CI . The second packet received at timing t = t20 takes 
the time D2 for decoding processing, but all the data cannot be decoded within this time because D2 > (t40-t20). 
[0074] Fig. 25A shows an entire encoded stream which should be originally decoded, and Fig. 25B shows data 
decoded partway owing to a short processing time. In this example, data can only be decoded up to the intemiediate 
bit plane BitS-2 in the intermediate subband HH2. Therefore, the image of this frame becomes smaller in image size 
50 than other frames. 

[0075] Fig. 24C is a view for explaining the image size in 2x-speed playback. The image size of the second frame 
is smaller than an imago size in nomrial playback in Fig. 24B. In practice, the image size rarely sequentially changes 
within one scene. The image has a small image size in a scene having a large code amount, like Fig. 24D, and a 
standard image size in a scene having a small code amount. Even if the image size changes, the number of frames 
to be played back does not decrease, and a smooth motion can be reproduced similarly to normal playback in Fig. 
248. In this manner, if the processing time is short, only decoded data are written in the memory to guarantee the 
image quality of at least the decoded data without decimating frames. 

[0076] Fig. 27A shows 4x-speed playback. As for the first packet received at timing t = tOO. all the data cannot be 
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decoded wrtthin the lime D1. and only data which can be decoded till the next packet reception timing t = t10 are 
displayed for the time CI . This also applies to D2. D3, and D4. In this example, since all the four packets cannot be 
decoded within their processing limes, the images of the frames are displayed with a small size, as shown in Fig. 27B. 
However, the number of frames to be played back does not decrease even at a 4x speed, so a smooth motion can be 
5 reproduced. 

[0077] As described above, image continuity can be maintained by the above processing for each packet though 
the size changes between frames. 

[Fifth Embodiment) 

10 

[0078] The fourth embodiment changes the image size and maintains the number of frames to be played back. The 
fifth embodiment limits the image size to be changed and can guarantee a predetermined image size, 
[0079] A decoding apparatus in the fifth embodiment has the same arrangement as that in Fig, 1 , Decoding process- 
ing in the fifth embodiment will be explained with reference to the flow chart of decoding processing of the decoding 
'5 apparatus in the fifth embodiment shown in Fig. 26. Note that a description of the same processing as in the flow chart 
shown in Fig. 23 will be omitted. Assuming that processing of step S2301 has already ended in processing of step 
S2601 , processing of step S2601 is not shown In Fig. 26, If the end of decoding of the current data is determined in 
step S2602, the same processing as processing from steps S2305 to S2311 is executed. 

[0080] Whether the processing lime has reached the limit lime is checked in step S2603. If Yes in step S2603, 
20 whether the image size of decoded data is larger than the smallest image size set in advance is determined in step 
S2604. The smallest image size is set in advance such that ail data must be decoded up to the subband LL or data 
must be decoded up to a size of LL + HL2 + LH2 + HH2. The set smallest image size is preferably set in accordance 
with the processing performance of the decoding unit, which will be described later. If Yes in step S2604, the flow 
advances to step S2605, and the decoding result is written in a memory 104a. 
25 [0081] The following processing is the same as processing in step S2310 and subsequent steps in Fig. 23. That is, 
the flow waits for the reception timing of the next frame data in step S2606. If the next frame data exists in step S2607, 
the next frame data is set as current frame data in step S2608, and decoding processing continues. If No in step S2607, 
the current frame data is determined as a final frame, and a series of processes end. 

[0082] If No in step S2604, the flow advances to step S2B09 and waits for the reception timing of the next, frame 
50 data. In step S261 0. whether the next frame data exists is checked. If Yes in step S261 0, the flow shifts to step S261 1 . 
and the limit time is changed. The limit time is changed by newly adding time management information within the packet 
header. That is. the current frame is processed within the total time of the processing time of the current frame and the 
processing time of the next frame. In step S2612, the next frame data is discarded, and data in a decoding buffer 1 02a 
is held. 

35 [0083] This relationship will be explained by exemplifying 4x-speed playback in Figs. 28A and 28B. In decoding 
processing of the first frame in Fig. 28A. all encoded data cannot be decoded, but the image size is larger than the 
smallest image size, and the first frame image is displayed for the time C1 . In decoding processing of the second frame, 
data cannot be decoded even with the smallest image size within the limit time D2, so the decoding time of the second 
frame is assigned up to the decoding limit time t of the third frame = t30. The display time is also prolonged to C2 + 

40 C3, which corresponds to two frames. In other words, the third frame image is not displayed, but the image size of the 
second frame image is held. 

[0084] When the smallest image size is not guaranteed, the frame frequency (the number of frames displayed per 
second) increases as the speed of high-speed playback increases. In some cases, the image size is preferably main- 
tained even by decimating frames in terms of the processing perfomnance of the decoding unit, the display perfonmance 

"ts of the display, and visual effects. The data amount per frame and display frame rate have the relationship shown in 
Fig. 31 . To increase the display frame rate, the data amount per frame must be decreased. To Increase the data amount 
per frame, the display frame rate must be decreased. The data amount and image quality have a relationship as shown 
in Fig. 30. In this case, decreasing the data amount per frame decreases the image size. The smallest image size is 
set by balancing the smoothness of the frame motion and the image size of each frame. As a simple implementation 

so means, the relationship between the playback speed of high-speed playback and the smallest image size is held as a 
table. Fig. 33 shows an exanriple of the table. In this example, the full image size is guaranteed up to a 3x speed, and 
a 1/4 image size is guaranteed up to a 6x speed. For a speed higher than the 6x speed, a 1/8 image size is guaranteed. 
[0085] As described above, the decoding apparatus of the fifth embodiment always manages the smallest image 
size of a frame to be decoded and always generates an image with a predetemnined or higher image size. 

55 

[Sixth Embodiment] 

[0086] The third embodiment prevents the image at the same tile position from successively failing in decoding. The 
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sixth embodiment gives high priority to tiles around the center of the screen so as to obtain an image with a stable si7e 
around the center of the screen. 

[0087] A decoding apparatus in the sixth embodiment has the same arrangement as that in Fig. 1 . Decoding process- 
ing in the sixth embodiment will be explained with reference to the flow chart of Fig. 29. A description of the same 
processingas that in the flow chart shown in Fig. 2 will be omitted. In the case, the processing unit is not a frame but a tile. 
[O088] The first band of the first tile in the current frame is processed for each plane (loop of steps S2903 -> S2904). 
After processing of all the planes ends, processing is repeated for the next band (loop of steps S2905 -> S2906). After 
processing of all the bands ends, processing is repeated for the next tile (loop of steps S2907-> S2908). This processing 
is repeated for all the tiles in the frame. 

[0089] In processing in units of tiles, the image size of the tile may be localized owing to the difference in data amount 
between tiles on the screen in high-speed playback. This is prevented by giving priority to the playback position of a 
tile image and decoding tiles in descending order of priority. A tile near the center of the screen is given high priority, 
and a tile at the end of the screen is given low priority. Accordingly, a stable-size image can be attained near the center 
of the screen. 

[0090] As described above, decoding processing of the decoding apparatus in the sixth embodiment enables de- 
coding the image at the same position with a stable size by giving priority to the decoding tile position in playing back 
a decoded frame. 
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[Other Embodiment] 

[0091] The following description concerns a case wherein the decoding apparatus shown in Fig. 1 according to the 
above embodiments is implemented by using a gen era I -purpose personal computer. 

[0092] Fig. 34 is a block diagram showing the hardware arrangement of a personal computer 50 functioning as the 
decoding apparatus shown in Fig. 1 . As shown in Fig. 34, the personal computer 50 comprises the following arrange- 
ment in addition to a CPU 1 for controlling the overall system, a ROM 2 which stores a boot program and the like, and 
a RAM 3 functioning as a main memory. 

[0093] An HDD 4 is a hard disk device. A VRAM 5 is a memory where image data to be displayed is mapped. Image 
data or the like can be mapped in the VRAM 5 and displayed on a CRT 6. Reference numeral 7 denotes a keyboard 
and mouse for performing various settings. 

[0094] As shown in Fig. 34. the HDD 4 stores an OS, the tables in Figs. 32 and/or 33, and a decoding program 
corresponding to a flow chart shown in at least one of Figs. 2. 5. 20, 23, 26, and 29. The decoding program also includes 
the functions of the DEMUX unit 101, decoding units 103a and 103b. and sync controllers 1 06a and 106b in Fig. 1. 
[0095] The RAM 3 also functions as the decoding buffers 102a and 102b and the memories 104a and 104b. The 
function of the output units 105a and 105b is implemented by the VRAM 5 and CRT 6. Designation of the playback 
speed, setting of the lowest image quality in the second embodiment, and/or setting of the smallest image size in the 
fifth embodiment is input via the keyboard and mouse. 

[0096] The OS and decoding program stored in the HDD 4 is loaded to the RAM 3 upon tuming on the power supply, 
and executed by the CPU 1 . The execution speed of decoding processing in this an-angement greatly depends on the 
processing perfomnance of the CPU 1 . It is, therefore, preferable to create a table (see Fig. 32) representing the rela- 
tionship between the playback speed and the lowest image quality so as to set the lowest image quality in accordance 
with the processing perfomiance of the CPU 1 in the second embodiment described above. Similarly, it is preferable 
to create a table (see Fig. 33) representing the relationship between the playback speed and the smallest image size 
so as to set the smallest image size in accordance with the processing performance of the CPU 1 in the fifth embodiment 
described above. 

[0097] This an-angement allows the personal computer to function as the decoding apparatus according to the 
present invention and perform processing of the above-described embodiments. 

[0098] The present invention may be applied to a system constituted by a plurality of devices (e.g., a host computer, 
interface device, reader, and printer) or an apparatus comprising a single device (e.g., a copying machine or facsimile 
apparatus). 

[0099] The object of the present invention is also achieved when a storage medium (or recording medium) which 
stores software program codes for realizing the functions of the above-described embodiments is supplied to a system 
or apparatus, and the computer (or the CPU or MPU) of the system or apparatus reads out and executes the program 
codes stored in the storage medium. In this case, the program codes read out from the storage medium realize the 
functions of the above-described embodiments, and the storage medium which stores the program codes constitutes 
the present invention. The functions of the above-described embodiments are realized not only when the computer 
executes the readout program codes, but also when the operating system (OS) running on the computer perfonns part 
or all of actual processing on the basis of the instructions of the program codes. 

[01 00] The functions of the above-described embodiments are also realized when the program codes read out from 
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the storage medium are written in the memory of a function expansion card inserted into the computer or the memory 
of a function expansion unit connected to the computer, and the CPU of the function expansion card or function ex- 
pansion unit performs part or all of actual processing on the basis of the instructions of the program codes. 
[0101] When the present invention is applied to the storage medium, the storage medium stores program codes 
5 corresponding to the above-mentioned flow chart (shown in at least one of Figs. 2, 5, 20. 23, 26, and 29). 

[01 02] As has been described above, the present invention can eliminate visual flaws between frames even if several 
frames cannot be decoded. 

[0103] The present invention is not limited to the above embodiments and various changes and modifications can 
be made within the spirit and scope of the present invention. Therefore, to apprise the public of the scope of the present 
10 invention, the following claims are made. 



Claims 

15 1 . A decoding apparatus for decoding an input signal which includes a code stream and data having time management 
information about decoding of the code stream, comprising: 

buffering means for buffering the input signal; 

decoding means lor reading out the input signal from said buffering means, decoding the input signal, and 
20 writing decoded data in a predetermined memory; 

output means for outputting the decoded data from the predetermined memory; and 

control means for controlling said buffering means and said decoding means on the basis of the time man- 
agement information. ». 

25 2. The apparatus according to claim 1 , wherein said control means checks a time necessary for decoding from the -^^ 
time management information, checks whether processing ends within a limit time, and when the time reaches 
the limit time before processing of all data ends, controls said decoding means to write decoded data in the pre- 
determined memory. ./-^ 

30 3. The apparatus according to claim 2. wherein when the time reaches the limit time before processing of all data . 

ends, said control means checks whether the decoded data have an image quality not lower than a preset lowest - . 
image quality, if the image quality is not higher than the lowest image quality, newly adds time management infer- \ 
mation to add a processing time of next data to a processing time of current data, and discards the next data. 

35 4. The apparatus according to claim 1 . wherein said decoding means reads out data from said buffering means in t; 

accordance with priority set by said control means, decodes the data, and writes the data at a position having the r 
same priority in the predetemnined memory. 

5. The apparatus according to claim 4, wherein said decoding means uses as a processing unit a tile obtained by 
40 dividing a frame at a predetermined size. 

6. The apparatus according to claim 4 or 5. wherein the priority set by said control means is determined to change 
for each frame so as to prevent priority at the same position from being the same between successive frames. 

-^5 7. The apparatus according to claim 5, wherein the priority set by said control means is detemnined to be high at a 
tile of each frame near a center of a screen. 

8. The apparatus according to claim 2, wherein when the time reaches the limit time before processing of all data 
ends, said output means outputs the decoded data written in the predetermined memory with an image size cor- 

50 responding to a size of the written data. 

9. The apparatus according to claim 2, wherein when the time reaches the limit time before processing of all data 
ends, said control means checks whether the decoded data have an image size not smaller than a preset smallest 
image size, if the image size is not larger than the smallest image size, newly adds time management infonmation 

55 to add a processing time of next data to a processing time of current data, and discards the next data. 

10. A method of controlling a decoding apparatus for decoding an input signal which includes a code stream and data 
having time management infonmation about decoding of the code stream, comprising: 
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a buffering step of buffering the input signal in a predetermined buffer; 

a decoding step of reading out the input signal from the predetermined buffer, decoding the input signal and 
writing decoded data In a predetermined memory; ' 
a output step of outputting the decoded data from the predetemiined memory and 

a control step of controlling at least one of the buffering step, the decoding step, and the output step on the 
basis of the time management information. 

The method according to claim 10. wherein in the control step, a time necessary for decoding is determined from 
the time management information, whether processing ends within a limit time is checked, and when the time 
reaches the limit time before processing of all data ends, the decoding step is controlled to write decoded data in 
the predetermined memory. 



The method according to claim 1 1 , wherein in the control step, when the time reaches the limit time before process- 
'"^ K 1 T^, decoded data have an image quality not lower than a preset lowest Image quality 

IS Checked, If the image quality Is not higher than the lowest image quality, time management information is newly 
added to add a processing time of next data to a processing time of current data, and the next data is discarded 

13. The method according to claim 10, wherein in the decoding step, data is read out from the buffering step in ac- 
cordance with priority set in the control step, decoded, and written al a position having the same priority in the 

20 predetennined memory. ^ » n'c 

14. The method according to claim 1 3, wherein in the decoding step, a tile obtained by dividing a frame at a predeter- 
mined size is used as a processing unit. 



15. The niethod according to claim 13 or14. wherein the priority set in the control step is determined to change for 
each frame so as to prevent priority at the same position from being the same between successive frames. 

16. The method according to claim 14, wherein the priority set in the control step is determined to be high at a tile of 
each frame near a center of a screen. 



process- 



1 7. The method according to claim 11 . wherein in the output step, when the time reaches the limit time before 
ing of all data ends, the decoded data written in the predetemiined memory are output with an image size corre- 
sponding to a size of the written data. 

1 8. The iTiethod according to claim 1 1 . wherein in the control step, when the time reaches the limit time before process- 
ing of all data ends, whether the decoded data have an Image size not smaller than a preset smallest image size 
IS determined, if the image size is not largerthan the smallest image size, time management infomiation is newly 
added to add a processing time of next data to a processing time of current data, and the next data is discarded 

19. A storage niedium whfch stores program codes functioning as a decoding apparatus for decoding an Input signal 
which includes a code stream and data having time management information about decoding of the code stream 

comprising: ^ , 



45 



so 



55 



a program code of the buffering step of buffering the input signal in a predetermined buffer- 

a program code of the decoding step of reading out the input signal from the predetennined buffer decodinq 

the input signal, and writing decoded data In a predetermined memory; 

a program code of the output slep of oulpuUing the decoded data from' the predetermined memory and 
a program code of the control step of controlling at least one of the buffering step, the decoding step and the 
output step on the basis of the time management infonnation. 

20. The apparatus according to claim 3. further comprising lowest image quality setting means for setting the lowest 
imaae aua itv. ^ 



21. 



The apparatus according to claim 20, wherein said lowest image quality setting means sets the lowest image 
quality in accordance with processing perfonnance of said decoding means. 

InlTrntf "k ^"^f"^ ^'^^"^ 21. Wherein said lowest image quality setting means sets the lowest image 
quality on the basis of relationship between the processing performance of said decoding means and a designated 
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playback speed. 

23. The apparatus according to claim 22, wherein 

5 the apparatus further comprises a first table representing the relationship between the playback speed and 

the lowest image quality, and 

said lowest image quality setting means sets the lowest image quality corresponding to the designated play- 
back speed with reference to the first table. 

^0 24. The apparatus according to claim 9, further comprising smallest image size setting means for setting the smallest 
image size. 

25. The apparatus according to claim 24, wherein said smallest image size setting means sets the smallest image 
size in accordance with processing perfomnance of said decoding means. 

15 

26. The apparatus according to claim 25. wherein said smallest image size setting means sets the smallest image 
size on the basis of relationship between the processing performance of said decoding means and a designated 
playback speed. 

20 27. The apparatus according to claim 26, wherein 

the apparatus further comprises a second table representing the relationship between the playback speed 
and the smallest image size, and 

said smallest image size sotting means sets the smallest image size corresponding to the designated playback 
25 speed with reference to the second table. 

28. The method according to claim 12, further comprising the lowest image quality setting step of setting the lowest 
image quality. 

30 29. The method according to claim 28, wherein in the lowest image quality setting step, the lowest image quality is 
set in accordance with processing perfomnance of the decoding step. 

30. The method according to claim 29, wherein in the lowest image quality setting step, the lowest image quality is 
set on the basis of relationship between the processing perfomnance of the decoding step and a designated play- 

35 back speed. 

31. The method according to claim 30, wherein In the lowest innage quality setting step, the lowest image quality 
corresponding to the designated playback speed is set with reference to a first table representing the relationship 
between the playback speed and the lowest image quality. 

40 

32. The method according to claim 1 8, further comprising the smallest image size setting step of setting the smallest 
image size. 

33. The method according to claim 32, wherein in the smallest image size setting step, the smallest image size is set 
^5 in accordance with processing perfomnance of the decoding step. 

34. The method according to claim 33, wherein in the smallest image size setting step, the smallest image size is set 
on the basis of relationship between the processing performance of the decoding step and a designated playback 
speed. 

so 

35. The method according to claim 34, wherein in the smallest image size setting step, the smallest image size cor- 
responding to the designated playback speed is set with reference to a second table representing the relationship 
between the playback speed and the smallest image size. 

55 36. A computer program containing processor implementable code for performing a method according to at least one 
of clainns 10-18 and 28-35. 
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